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Disclaimers 


This topic is bigger than fits in this lecture 

All code shown will be python 

Techniques should work in multiple languages 

Have build/worked with versions of these approaches 

• bash 

• python 

• ruby 

• Perl 

• C 

• Java 

• C# 



What is the purpose of testing? 


Make Testing Great Again! 

Writing tests is extra work 

It has to be easy for people to write tests for new functions 
It has to be easy for people to run tests to feel worth writing tests 
Ideally most tests run on people's laptops 
Test failures should be repeatable 


You don't have to go it alone 


Java -junit 
Python - unittest 
Perl-Test.pm 

C - autotest (part of autoconf / automake) 

Nearly every language environment has test framework to get running 
with 



The simplest Test File: test_mystuFF.py 


inport unittest 

class TestMyTest(unittest.TestCase) : 
def setl)p(self ) : 

def test_something(self ) : 

self .assertEqual(a, b) 

def test_some_other_thing(self ) : 

self . assertRaises( exception . Foo, 
somefunc, argl, arg2) 

def tearDown(self ) : 


Naming is magical, and important 

• Filename must start with test * 

• Test classes must start with Test.* 

• Test Functions must start with test.* 

run order: setup -» test_Foo -> tearDown 
For every test 

all tests run in a single process, global state 
changes will not automatically be reset 

run order oF test_Foo vs. test_Foo2 is not 
deFined, plan accordingly 

all test systems will have caveats, read up on 
them 




Actually running tests 


What has to be installed? 

What are the dependencies? 
Testing different environments? 



Is it even software? 


def sum(*args): 
res = 0 

for arg in args: 
res += agr 
return res 


• Compilers help in compiled 
languages 

• In dynamic languages, you need 
linters 

• for python: flake8 / pylint 




Writing testable code 


What's perfectly testable code? 

• Limited inputs 

• Limited branches 

• No side effects - no state saving, no communication to other systems 

• AKA -LISP 


This is < 5% of all interesting software 
What do we do for the other 95%? 


Fakes 


& 


Build a fake implementation of 
an abstraction layer 

Load and run for an entire test 
class 

Pros: 

• Easy to replace in testing 

• Can respond with anything you want, 
including odd errors that are hard to get 
in production 

Cons: 

• Only as accurate as you make it 

• Can easily become a complex simulator 
that has it's own bugs 


Mocks 


Dynamically replace function 
calls 

Typically done per test method 
Pros: 

• Very precise isolation 

• Don't need to build a whole fake 

• Really easy to do fault injection 

Cons: 

• Can very easily drift from real behavior 

• Can make refactoring hard 


Fakes 


compute. api 

compute. api 

compute. api 


compute. manager 

compute. manager 

compute. manager 


virt. driver 

virt. driver 

virt. driver 


virt.driver.libvirt 

virt.driver.libvirt 

fake virt 


libvirt 


fakelibvirt 






Mocking in action 



return_value=ll) 

def test_validate_auto_or_none_network_request_old_cornputes(self , rnock_get_ver) : 

Tests that the network request is nulled out when the minimum 

nova-compute is not running new enough code to support 'auto'. 

ii ii ii 

req_nets = objects. NetworkRequestList( 

objects=[objects.NetworkRequest(network_id= ' auto ' )] ) 

self .assertIsNone( 

self . controller. _validate_auto_or_none_network_request(req_nets)) 



Reference to 
mock 



mock_get_ver .assert_called_once_with(mock.ANY, ' nova-compute ' ) 



test that this was called 
exactly once with these 
parameters 





Integration Testing 

• Assemble the whole system into some workable whole 

• Run some representative set of tests on it 

• Ensure that all the individual working parts build a working whole 



/Bk Bill Sempf £ ±* Follow 

. sempf 


QA Engineer walks into a bar. Orders a beer. 
Orders 0 beers. Orders 999999999 beers. 
Orders a lizard. Orders -1 beers. Orders a 
sfdeljknesv. 
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2016-10-30 16:52:19,208 [DEBUG] root: cclass 'arwn. vendor. RFXtrx.SensorEvent'> device=[<class 

'arwn. vendor. RFXtrx.RFXtrxDevice'> type= ' PCR800 ' ld= ' 96 : 00 ' ] values=[ (' Battery numeric', 9), ('Rain Rate (mm/hr)', 

0.99), ('Rain Total (mm)', 422.7), ('Rssi numeric', 6)] 

2016-10-30 16:52:22,467 [DEBUG] root: <class 'arwn. vendor. RFXtrx.ControlEvent ' > device=[<class 

'arwn. vendor. RFXtrx.LightingDevice'> type= 1 FlomeEasy EU' id='2aa82aa:ll' ] values=[( 'Command ' , 'Set group level'), ('Dim 
level', 68), ('Rssi numeric', 2)] 

2016-10-30 16:52:22,468 [DEBUG] PidFile: <pid.PidFile object at 0x762bble8> closing pidfile: 

/home/sdague/code/arwn/arwn. pid 

2016-10-30 16:52:22,469 [ERROR] root: Something went wrong! 

Traceback (most recent call last): 

File "/home/sdague/code/arwn/.venv/local/lib/python2.7/site-packages/arwn-1.0.0-py2.7.egg/arwn/cmd/collect.py" , line 
80, in main 

event_loop(config) 

File "/home/sdague/code/arwn/.venv/local/lib/python2.7/site-packages/arwn-1.0.0-py2.7.egg/arwn/cmd/collect.py" , line 
67, in event_loop 

dispatcher . loopforever( ) 

File "/home/sdague/code/arwn/.venv/local/lib/python2.7/site-packages/arwn-1.0.0-py2.7.egg/arwn/engine.py" , line 186, 
in loopforever 

packet . from_packet (event . device . pkt ) 

File "/home/sdague/code/arwn/.venv/local/lib/python2.7/site-packages/arwn-1.0.0-py2.7.egg/arwn/engine.py", line 76, in 
from_packet 

self.bat = packet. battery 

AttributeError: 'Lighting2' object has no attribute 'battery' 

2016-10-30 16:52:22,473 [DEBUG] PidFile: <pid.PidFile object at 0x762bble8> closing pidfile: 

/home/sdague/code/arwn/arwn. pid 


The journey of a thousand miles begins with a single step 





phue.py 


Python interface to Hue light bulbs 
Constraints: 

• Single Pile 

• No external requirements 

• Has to work in python 2 & 3 

No existing testing at all 



First test 


import testtools 
import phue # noqa 

class Testlmport(testtools.TestCase) : 

def test_import_works(self ) : 
pass 




Test runner-tox.ini 


[tox] 

envlist = pep8, py27, py35 
skip_nissing_interpreters = True 

[testenv] 
setenv = 

LANG=en_US.UTF-8 
PYTHONPATH = {toxinidir} 
commands = 

py.test -v - -timeout=30 - -duration=10 --cov=phue --cov-report html {posargs} 
deps = 

- r{ toxinidir}/ test- requirements . txt 

[testenv:pep8] 
deps = flake8 
basepython = python3 
commands = 

flake8 phue.py 

[flake8] 
ignore = E501 

exclude = .venv, .git, .tox,dist, doc, *lib/python*,*egg, build 



What's this software actually look like? 



lookup username 


phue_conf 


self.requestCPOST', '/api') 


Hue Hub 


.lights 


self.requestCGET', '/api/Susername/lights') 


phue. Light 


V 


phue. Light 


phue. Light 


Hue Hub 




.bridge 
. light Jd 

.name 

.on 

■xy 


self. bridge. request('GET\ yapi/$username/lights/$lightjcr) 






def request(self , node='GET', address=None, data=None): 

Utility function for HTTP GET/PUT requests for the API""" 

connection = httplib.HTTPConnection(self .ip, timeout=10) 

try: 

if node == 'GET' or node == 'DELETE': 
connection . request (node, address) 
if node == 'PUT' or node == 'POST': 

connection . request(node, address, data) 

logger .debug( "{0} {1} {2}" . fornat(node, address, str(data))) 

except socket .tineout: 

error = "{} Request to {}{} tined out . " .fornat(node, self. ip, address) 

logger .exception (error) 

raise PhueRequestTineout(None, error) 

result = connection .getresponse() 
connection . close( ) 
if PY3K: 

return json.loads(str(result . read() , encoding= ' utf -8 ' ) ) 
else: 

result_str = result . read( ) 
logger .debug (result_str) 
return json.loads(result_str) 





class TestRequest(testtools .TestCase) : 
def setUp(self ) : 

super(TestRequest , self ) . setUp( ) 
self. hone = fixtures .TempHomeDir( ) 
self .use Fixture (self .hone) 

def test_register(self ) : 

"""test that registration happens autonatically during setup, 
confnane = os . path . join(self . hone. path, ' . python_hue ' ) 
with nock. patch( "phue. Bridge. request" ) as req: 

req. return_value = [{'success': {'usernane': 'fooo'}}] 
bridge = phue. Bridge(ip="10. 0.0.0") 
self .asser t Equal ( bridge. config_file_path, confnane) 

# check contents of file 
with open(confnane) as f: 

contents = f.read() 

self .assertEqual(contents, '{"10.0.0.0": {"usernane": 

# nake sure we can open under a different file 
bridge2 = phue. Bridge(ip="10. 0.0.0" ) 
self .asser t Equal (bridge2 .usernane, "fooo" ) 

# and that we can even open without an ip address 
bridge3 = phue.BridgeQ 

self .asser t Equal (bridge3 .usernane, "fooo" ) 
self .asser t Equal (bridge3 .ip, "10.0.0.0") 



All request calls return 
same results 


"fooo"}}' ) 


What's this software actually look like? 



lookup username 


phue_conf 


self.requestCPOST', '/api') 


Hue Hub 


.lights 


self.requestCGET', '/api/Susername/lights') 


phue. Light 


V 


phue. Light 


phue. Light 


Hue Hub 




.bridge 
. light Jd 

.name 

.on 

■xy 


self. bridge. request('GET\ yapi/$username/lights/$lightjcr) 






def request(self , node='GET', address=None, data=None): 

Utility function for HTTP GET/PUT requests for the API""" 

connection = httplib.HTTPConnection(self .ip, timeout=10) 

try: 

if node == 'GET' or node == 'DELETE': 
connection . request (node, address) 
if node == 'PUT' or node == 'POST': 

connection . request(node, address, data) 

logger .debug( "{0} {1} {2}" . fornat(node, address, str(data))) 

except socket .tineout: 

error = "{} Request to {}{} tined out . " .fornat(node, self. ip, address) 

logger .exception (error) 

raise PhueRequestTineout(None, error) 

result = connection .getresponse() 
connection . close( ) 
if PY3K: 

return json.loads(str(result . read() , encoding= ' utf -8 ' ) ) 
else: 

result_str = result . read( ) 
logger .debug (result_str) 
return json.loads(result_str) 





LIGHTS1 = { 

u ' 1 ' : {u 'manufacturer-name' : u' Philips', 
u'modelid': u'LCTGGl', 
u'name': u' Living Room Bulb', 
u' state': {u 1 alert': u'none', 
u'bri' : 254, 
u' colormode' : u'xy' , 
u ' ct ' : 382, 
u' effect' : u'none' , 
u ' hue ' : 14665, 
u 'on ' : True, 
u ' reachable' : True, 
u ' sat ' : 156, 

u'xy' : [0.4677, 0.4121]}, 
u ' swversion ' : u '5. 23.1. 13452 ' , 
u'type': u' Extended color light', 
u'uniqueid' : u ' 00 : 17 : 88 : 01 : 00 : dl : fd : 53 - 0b ' } , 

} 

RESP = dict(GET=dict(), P0ST=dict(), PUT=dict(), 
DELETE=dict( ) ) 

RESP[ ' GET ' ] [ ' /api/username/lights/ ' ] = LIGHTS1 
for key, value in LIGHTSl.itemsQ : 

RESP[ ' GET ' ] [ ' /api/username/lights/%s ' % key] = \ 
LIGHTSl[key] 


class FakeHTTP(object) : 

def init (self, *args, **kwargs): 

super(FakeHTTP, self). init () 

self. call = None 

def request(self , mode, addr, data=None): 
self. call = Request(mode, addr, data) 

def getresponse(self ) : 

data = samples. RESP[self. call. mode] [self .call. addr] 
return StringIO(dump(data)) 

def close(self): 
pass 


class TestLights(testtools.TestCase) : 

def setUp(self): 

super(TestLights, self).setUp() 
self .useFixture( 

fixtures.MonkeyPatch(httplib, fakes . FakeFITTP)) 
self. bridge = phue.Bridge( 

ip=" 10. 0.0.0" , username=" username" ) 

def test_get_lights(self ) : 

lights = self .bridge. get_light_objects( 'id' ) 

self .assertEqual(lights[l] .name, "Living Room Bulb") 






Keeping track oF progress 


Coverage for phue.py : 34% 


673 statements 


228 run 


445 missing 
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U ! /us r /bin/python 
it coding: utf-8 -*- 


phue by Nathanael Lecaude - A Philips Hue Python library 

Contributions by Marshall Perrin, Justin Lintz 

https : //github . com/studioimaginaire/phue 

Original protocol hacking by rsmck : http://rsmck.co.uk/hue 

Published under the MIT license - See LICENSE file for more details. 

"Hue Personal Wireless Lighting" is a trademark owned by Koninklijke Philips Electronics N.V. 
I am in no way affiliated with the Philips organization. 


import json 
import logging 
import os 
import platform 
import sys 
import socket 

if sys.version_info[0] > 2: 

PY3K = True 
else: 

PY3K = False 
if PY3K: 

import http. client as httplib 
else: 

import httplib 

logger = logging. getLogger ( 'phue' ) 


if platform. system( ) == 'Windows': 
USER_H0ME = 'USERPROFILE' 
else: 

USER_H0ME = 'HOME' 
version = '0.9' 



Continuous Integration 


• Things that happen automatically will happen more oFten 

• IF soFtware is easy to test, it's easy to ship 

• Many Cl tools out there 

• Jenkins 

• Zuul 

• Circle Cl 

• etc... 


Travis Cl X Bi og status 


Help 


Sean Dague 


Search all repositories 


Q. 


sdague / phue Q 


build unknown 


My Repositories + 


Current Branches Build History Pull Requests 


X home-assistant/home-assistair #14016 


•y tests build framework for testing more complex API interactions 


-o- #6 passed 


© Duration: 18 min 23 sec 
[f] Finished: 13 minutes ago 


This builds a framework for having a set of canned test responses 
based on HTTP Requests. It does this by monkey patching out 

HTTPConnection, and redirecting it into a static multi level 


•y sdague/arwn # 44 

© Duration: 1 min 57 sec 
[13 Finished: a day ago 

y sdague/rxv # 2 


Commit 7026ad9 
Compare d8e81a6..7026ad9 

Q Sean Dague authored and committed 


(fy Elapsed time 40 sec 
© Total time 1 min 4G sec 

[1] 5 days ago 


© Duration: 1 min 38 sec 
[U Finished: 3 days ago 

X sdague/home-assistant 

© Duration: 14 min 57 sec 
[13 Finished: 3 days ago 


Build Jobs 


■y 

#6.1 

6 

</> Python: 2.7 

® TOXENV=py27 

*y 

#6.2 

a 

</> Python: 3.5 

® TOXENV=py35 

*y 

#6.3 

a 

</> Python: 2.7 

® T0XENV=pep8 


•y sdague/phue #6 

© Duration: 1 min 46 sec 
[13 Finished: 5 days ago 


© 36 sec 
© 39 sec 
© 31 sec 


More options 


O Restart build 




The Scientific Method as an Ongoing Process 



By ArchonMagnus - Own work, CC BY-SA4.0, https://commons.wikimedia.org/w/index.php?curid=42164616 


Questions? 


